Expected Utility Estimation via Cross-Validation

We discuss practical methods for the assessment, comparison and selection of complex hierarchical Bayesian models. A natural way to assess the goodness of the model is to estimate its future predictive capability by estimating expected utilities. Instead of just making a point estimate, it is important to obtain the distribution of the expected utility estimate in order to describe the associated uncertainty. We synthesize and extend the previous work in several ways. We give a unified presentation from the Bayesian viewpoint emphasizing the assumptions made and propose practical methods to obtain the distributions of the expected utility estimates. We discuss the properties of two practical methods, the importance sampling leave-one-out and the k-fold cross-validation. We propose a quick and generic approach based on the Bayesian bootstrap for obtaining samples from the distributions of the expected utility estimates. These distributions can also be used for model comparison, for example, by computing the probability of one model having a better expected utility than some other model. We discuss how the crossvalidation approach differs from other predictive density approaches, and the relationship of cross-validation to information criteria approaches, which can also be used to estimate the expected utilities. We illustrate the discussion with one toy and two real world examples.